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Abstract 

Artificial spike-based computation, inspired by models of computation in the central 
nervous system, may present significant performance advantages over traditional meth- 
ods for specific types of large scale problems. This paper describes very simple network 
architectures for /c-winners-take-all and soft-winner-take-all computation using neural 
oscillators. Fast convergence is achieved from arbitrary initial conditions, which makes 
the networks particularly suitable to track time-varying inputs. 



1 Introduction 

The discovery of synchronized oscillations in the visual cortex and other brain regions has 
triggered significant research in artificial spike-based computation jH IHl CHI Ull HH UZl UH 
HH 123 1^21 ESI- While neurons in the central nervous system are about six orders of 
magnitude "slower" than silicon-based elements, in both elementary computation time and 
signal transmission speed, their performance in networks often compares very favorably with 
their artificial counterparts even when reaction speed is concerned. In a sense, evolution 
may have been forced to develop extremely efficient computational schemes given available 
hardware limitations. 



In a recent paper [M], we proposed new models for two common instances of such neural 
computation, winner-take- all and coincidence detection, featuring fast convergence and 0{n) 
network complexity. We saw that both computations could be achieved using a similar archi- 
tecture, using global feedback inhibition in the first case, and global excitation in the second. 
In this paper, we further extend this computational architecture to fc-winners-take-all and 
soft-winner-take-all. 

Fast Winner-take-all (WTA) computation in [SI] is based on the FitzHugh-Nagumo model, 
a well-known simplified version of the classical Ho dgkin- Huxley model. Compared to previous 



^To whom correspondence should be addressed. 



1 



WTA networks Pl El El El El EH], it has significant computational advantages. The network's 
initial states can be set arbitrarily, and convergence is guaranteed in at most two spiking 
periods, with a high computation resolution. The network's complexity is linear in the number 
of inputs and its size can be adjusted at any time during the computation. As this paper shows, 
by modifying the starting point of the global inhibitory neuron's charging mode. A;- Winners- 
Take- All (fc-WTA) can be computed instead. By running the charging mode independently, 
soft- Winner- Take- All (soft-WTA) can be computed. Both extensions inherit the advantages 
of the original WTA network. 

After a brief review of the basic WTA network in section El A;- WTA computation and 
soft-WTA computation are studied in Sections El and El Brief concluding remarks are offered 
in Section El 

2 Winner- Take- All Network 

The WTA network in (MI is based on the FitzHugh-Nagumo (FN) model fTl l^l^ : 

V = v{a — v){v — 1) — w + I 
w = (3v — jw 

For appropriate parameter choices, there exists a unique equilibrium point for any given value 
of /, which is stable except for a finite range Ii < I < Ih where the system tends to a limit 
cycle. The steady-state value of v at the stable equilibrium point increases with /. 



Global inhibitory neuron 
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Figure 1: Diagram of the WTA network. There are n FN neurons receiving external inputs 
and a global inhibitory neuron monitoring the whole network. 

The network structure is illustrated in Figure d where n FN neurons receive external 
stimulating inputs Jj and a global inhibition z. The dynamics of the FN neurons (i = 1, . . . , n) 

are 

Vi = Vi{a - Vi){vi - 1) - Wi + li - z 
Wi = (3vi - 'jWi 
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The dynamics of the global inhibitory neuron is 



z = 



—kc {z — zq) charging mode 
—kdZ discharging mode 



(1) 



which starts charging if there is any FN neuron spiking in the network (i.e., if 3z, Vi > Vo 
for some given threshold vq) and switches to discharging if the state z is saturated. With a 
fast charging rate kc and a slow discharging rate kd, the network computes the largest input 
(corresponding to the only spiking FN neuron) in at most two periods. Initial conditions 
can be set arbitrarily and the computation resolution is very high. Detailed analysis and 
discussions can be found in i34i . 



K-WTA, a common variation of WTA computation where the output indicates for each neuron 
whether its input is among the k largest, has been studied in such fields as competitive learning, 
pattern recognition and pattern classification jSl El ISS ISHl EI] • As Maass argued in (THj , in 
principle a fc-WTA network can replace a two-layer threshold circuit to perform most standard 
nonlinear computational operations. 

Most k-WTA studies are based on steady-state stability analysis. Many models define 
the winners as the neurons with the largest initial states or require initial conditions 

to be set precisely jHTj, making the networks not well suited to time- varying inputs. Others 
adopt particular design methodologies [23 1211 tiut the network size or the number of winners 
is limited. K-WTA is also implemented in analog VLSI circuits which extend the elegant 
WTA model in ^^1 but inherit its low resolution limit as well. 

The neural network described in Section |21 can be easily extended to fc-WTA computation, 
where an FN neuron spikes if and only if its input is among the k largest. Indeed, as the 
global inhibition force decreases, the FN neurons enter the oscillation region rank-ordered by 
their inputs. Thus, while for WTA computation, the global inhibition neuron is charged after 
the first arrival, for fc-WTA computation the charging moment is simply modified to capture 
the k^^ arrival instead. 

To this effect, we augment the dynamics of the FN neuron with an additional state variable 
Ui (for simplicity, we shall still call FN neuron such a generalized element) 



where Uq is a constant saturation value and k^ the charging/discharging rate. The variable 
Q takes two values, namely it switches to whenever z approaches a saturation value zo, 




3 i^- Winners- Take-All Network 




Vi{a - Vi){vi - 1) - Wi + li - Ui - z 
(3vi - -fWi 
ku {Quq - Ui) 
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else it switches to 1 if Vi exceeds a given threshold vq. This make the dynamics of Ui a local 
self-inhibition, which starts charging if the basic FN neuron spikes and discharges whenever 
the global inhibitory neuron spikes. Note that the specific form of the dynamics of Ui can be 
more general, as long as the value of Ui varies between and uo, and the transition periods 
are very fast (which is satisfied here by choosing a large ku)- 

The dynamics of the global inhibitory neuron is the same as (P), except that we start its 
charging mode if any k FN neurons in the network spike. Such a moment can be captured by 
determining that Y17=i ^« approaches kuo- Thus, if any FN neuron spikes, it excites only the 
corresponding local inhibitory portion but has no effect on the rest of the network. If there 
are k local inhibitions turned on, the global inhibitory neuron is charged, which then releases 
all the local inhibitions and starts a new period. 

Compared to the WTA network in Section |21 the basic principle underlying the /c-WTA 
network described above is the same, exploiting the simple properties of the FN model. Thus, 
most of the computational advantages of the WTA network sue inherited by the /c-WTA 
extension. In particular 

• The initial conditions of the network can be set arbitrarily. 

• With appropriate parameters, the computation can be completed at most in two periods, 
where the first period is affected by the initial conditions but the k spiking neurons during 
the following periods are guaranteed to be those with the largest inputs. If the initial 
inhibitions are strong, the computation is completed in one period. 

• Since initial conditions are immaterial and the computation speed is very fast, the 
A;- WTA network is able to track time-varying inputs. Moreover, since the network 
complexity is 0{n), individual FN neurons can be added or removed at any time during 
the computation. 

• The inputs Jj should be lower-bounded by //, the lower threshold of the FN oscillation 
region. It should also be upper-bounded to set inhibition saturations, although the 
upper bound value is not restricted. 

• The computation resolution also follows that of the WTA network. It can be improved 
by decreasing the discharging rate k^, as well as the relaxation time of the FN neurons. 

• FN neurons receiving equal inputs behave identically, which means that the A;- WTA 
computation may generate more than k winners in this particular case. 

Example 3.1: The result is illustrated in simulation in Figure [2 with n = 10 and k = 3. The 
parameters of the FN neurons are set as q = 5.32, /3 = 3, 7 = 0.1 , with spiking threshold vq = 5 . 
The parameters of the local inhibition are uq = 160, = 100 . The inputs li are chosen randomly 
from 20 to 125. The parameters of the global neuron are zq = 240, kc = 100, fc^ = 1/40 . All 
initial conditions are chosen arbitrarily. The three spiking neurons after the first charging of the 
global neuron are those with the three largest inputs. 
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Note that the output frequency is determined mainly by the global neuron's dynamics and the 
value of the /c*^ largest input. It can be increased by increasing the global neuron's discharging 
rate after the first winner spikes so as to facilitate the other winners' spiking. D 



Example 3.2: Figure 01 illustrates a simulation result with n = 3 and k = 2. The parameters 
are the same as those in Example 13.11 The inputs keep varying and switch winning positions 
several times. The spiking neurons always track the two largest inputs. D 



4 Soft- Winner- Take- All 

Soft-WTA jini (or softmax) is another variation of WTA computation, where the outputs 
reflect the rank of all inputs according to their size. Although soft-WTA is a very powerful 
primitive 1201 in that it can be used to compute any continuous function, its "neural" 
implementation is complex. Recently, [21] studied soft-WTA as an optimization problem not 
based on a biologically plausible mechanism; presented a hardware model of selective vi- 
sual attention which lets the attention switch between the selected inputs, but whose switching 
order does not completely reflect the input ranks. In this section, we develop a simple neural 
network which computes soft-WTA very fast and generates spiking outputs rank-ordered by 
their inputs. 

Let k = n in the A;- WTA network described in Section IHl Then we get a pre-ordered 
spiking sequence in each stable period, since all the FN neurons enter the oscillation region 
and then spike rank-ordered by their inputs. However, such an n*^ arrival moment may not 
be measurable if the number of inputs n is unknown or time-varying. To avoid this problem 
and make the solution more general, we let the charging mode of the global inhibitory neuron 
start only if the inhibition z is lower than a given bound zio^j. The spikings of all the FN 
neurons in the network are guaranteed by the condition that 

^low ^ ~^ ^min 

where Ii is the lower bound of the oscillation region of the FN model and Imin is the minimum 
input value. 

Example 4.1 : Figure |1] illustrates the result in simulation with n = 10. The parameters are the 
same as those in Example 13.11 The inputs Ii are distributed uniformly between 80 and 120. The 
inhibition lower bound is z/o^ = 60 . Initial conditions are chosen arbitrarily. The computation 
is completed in the second period, during and after which the spiking times of the FN neurons are 
ranked by their inputs. D 

The simple soft-WTA network presented above inherits the main computational advan- 
tages of our WTA and fc-WTA networks. Initial conditions can be arbitrary, the computation 
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Figure 2: fc-WTA computation result of Example 13.11 with n = 10 and A; = 3. The plots are 
the time developments of (a) Vi of the neurons with the three largest inputs; (b) Ui of the 
neurons with the three largest inputs; (c) Vi of the other seven neurons; (d) Ui of the other 
seven neurons; (e) global inhibition z. The computation is completed in less than two periods. 
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Figure 3: fc-WTA computation result of Example 13.21 with n = 3 and k = 2. The inputs 
are not constant. The spiking neurons always track the two largest inputs. The plots are 
(a). states Vi versus time; (b). inputs versus time. Note that each vt in plot (a) corresponds 
to the li in plot (b) with the same line type (solid, dashed or dotted). 



is completed in at most two periods, network complexity is linear and neurons can be added 
or removed at any time. We expect it to be effective in many applications such as selec- 
tive attention, associative memory and competitive learning, and also to provide an efficient 
desynchronization mechanism for perceptional binding [TTH l?7| E21 ESI • 



5 Concluding Remarks 

Basic neural computations such as winner-take-all, A;-winners-take-all, soft-winner-take-all, 
and coincidence detection can all be implemented using a common architecture and biological 
plausible neuron models. Fast and robust convergence is guaranteed, and time- varying inputs 
can be tracked. Further research will study models of higher level brain functions, such as 
perception, based on the WTA networks and on general nonlinear synchronization mechanisms 
derived in j2HlEHl- 
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Figure 4: Soft-WTA computation result of Example 14 . 1 1 with n = 10. The plots are (a). states 
Vi versus time; (b). global inhibition z versus time. The initial conditions are chosen arbitrarily 
and the computation is completed in the second period. 
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